feat: per-step progress logging for the multi-start gradient searches - #1435
Merged
Merged
Conversation
The MultiStartGradient searches emitted two log lines for an entire run -- "Starting new ..." and "... sampling complete" -- leaving a user unable to tell a live fit from a hung one. Neither of the framework's progress channels reaches them: iterations_per_full_update makes the whole run a single chunk whose lone perform_update is (correctly) suppressed as duplicated work, and Fitness's quick-update counter is Python state mutated inside fitness.call, which these searches trace under jax.jit(jax.vmap(...)) -- so it runs once at trace time and never again. The samplers delegate to their own library's progress bar; a search that owns its step loop has to report for itself. Adds iterations_per_log (default 10) on AbstractMultiStartGradient, inherited by all four concrete searches. The line reports step, best log posterior, gain since the last line and live-start count, plus Prodigy's estim_lr -- its self-estimated step scale -- where the rule carries one. The first step always logs, since that line is what tells the user the XLA compile finished and stepping has begun. Two JAX-compile notices cover the single-point objective _broad_starts calls per draw and the vmapped/chunked one the step loop calls. Neither is reachable by the Fitness._jit/_vmap/_grad notices, because this search builds its transforms straight off fitness.call. All output is gated on the existing silence flag. Numerically inert: no traced operation is added, and the loop already forced a device sync every step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Unu8t9xqScV93XinRmx1Do
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1433.
af.MultiStartProdigy/MultiStartAdam/MultiStartADABelief/MultiStartLionemitted exactly two log lines for an entire run —"Starting new <rule> MultiStartGradient search (N starts, ...)"and, however many hours later,"<rule> MultiStartGradient sampling complete."Nothing in between, so a user could not tell a live 300-step fit from a hung one.Neither of the framework's usual progress channels reaches this search, which is why it needs its own:
iterations_per_full_updatedefaults to the never-sentinel, so the whole run is a single chunk whose loneperform_updateis (correctly) suppressed by_is_final_boundaryas duplicated work.Fitness's quick-update counter is Python state mutated insidefitness.call, which this search traces underjax.jit(jax.vmap(…))— it runs once at trace time and never again.The samplers dodge the problem by delegating to their own library's progress bar (emcee
progress=True, dynestyprint_progress); a search that owns its step loop has to report for itself.This adds a cadence-controlled progress line plus two JAX-compile notices, all gated on the existing
silenceflag.Why a log line and not a progress bar:
tqdmis only a transitive dependency, auto-convergence makesn_stepsa ceiling the search routinely stops short of (so a bar's ETA would mislead), and lines survive SLURM/HPC log capture where bars do not.Why the compile notices live here: the compile-message work in #1434 targets
Fitness._jit/_vmap/_grad, which these searches never use — they build their transforms straight offfitness.call(search.py:260-261). Without this, that fix would ship and these searches would still sit through their compile in silence. Two distinct compiles block a fresh run and both are covered: the single-point objective_broad_startscalls per draw, and the vmapped/chunked one the step loop calls. Wording is matched to #1434 rather than inventing a second dialect.API Changes
One additive, default-preserving constructor argument on
AbstractMultiStartGradient, inherited by all four concrete searches:iterations_per_log: int = 10. No existing signature, default or behaviour changes, and no symbol is removed or renamed — a caller that ignores it sees identical results, only new log output.See full details below.
Test Plan
python -m pytest test_autofit/→ 1614 passed, 1 skippedMultiStartProdigyfit on the 1D Gaussian, confirming the lines fire in a live run (output below)Real run (
n_starts=16,n_steps=120,iterations_per_log=10):Note the two compiles account for ~4.4s while all 120 steps take 0.3s — on this toy problem the compile is the wait, which is exactly why it needed a notice. The
d(Prodigy'sestim_lr) column ramps1e-06 → 1.19e+01while the fit is still finding its basin, then plateaus: previously invisible, and the one genuinely learning-rate-free diagnostic.Notes for review
AbstractMultiStartGradient, so all four subclasses get the line — the silence was a base-class problem, not a Prodigy-only one.updates:block would imply every search honours it (they don't) and wouldKeyErrorin any workspace whose config shadows that section — the trap hit in feat(autofit): multi-start gradient convergence results contract (phase 2) #1409. A plain constructor kwarg avoids both.search.py:361(np.isfinite(np.asarray(foms))), so the step/fom/alive fields are free.estim_lris an(n_starts,)device→host copy taken only on a logging step, which merely pulls forward the sync the next iteration would force anyway. Nothing traced is added.abstract_search.py/fitness.py). Branched off the samea50ba95b0;origin/mainhad not moved at PR-open.Full API Changes (for automation & release notes)
Added
AbstractMultiStartGradient.__init__(..., iterations_per_log: int = 10)— steps between progress lines on the search log. Inherited byMultiStartAdam,MultiStartADABelief,MultiStartLionandMultiStartProdigy. Serialised throughto_dict/from_dict, so a resumed search keeps the chosen cadence. Rejected at construction (viaAbstractSearch._check_step_count) if not a whole number ≥ 1, since a sub-1 cadence either raises on the modulo or logs every step.Changed Behaviour
_fit, where previously they were silent between start and completion. Emitted atlogging.INFOon the search's own logger and fully suppressed by the existingsilence=True; numerical results are unchanged.Migration
None required — the argument is optional and defaults to the new behaviour. Pass
silence=Truefor the previous, fully-silent output.Generated by the PyAutoLabs agent workflow.